SOURCE NOTE (consolidation): This is a draft/fragment. The flagship QDA-facing paper is now: Causal mapping as causal QDA.
Companion methods notes: Magnetisation; A simple measure of the goodness of fit of a causal theory to a text corpus.

opinionated AI coding

special advantages of causal QDA through the eyes of AI

• We often see evaluators and other researchers using AI for tasks like "list the main themes in this document" or even "list the main themes in this collection of documents". To be clear: we've all done it. there are times when it can be a useful time-saver. But the trouble with that is - it's massively sensitive to what one means by a theme. What do you mean by "theme"? - you can improve your prompt massively simply by narrowing the universe: - "Identify the main kinds of relationship issues mentioned" - The natural conclusion of this narrowing-down is reducing the generation problem to a categorisation problem. Categorisation is probably taking things too far, because you lose the advantage of any kind of identification of unexpected things. Wouldn't it be great if you could just have a generic instruction like "make sense of this document already. Just tell me what's going on, but not only as a summary, but also as a report which is to some extent representative of the different contributions from different sources or sections, and in such a way that it's somehow intersubjectively verifiable ?!?! There is such an instruction: it's called causal mapping. The chances of two independent coders achieving somewhat similar results are much bigger with this reformulation. This is anecdotal, I don't have a reference for it. There are two issues - chunk size and intersubjective verifiability. Ensemble agreement is the second-loop version of intersubjective verifiability. Rick D is going to love Workflows
• 1. We put the question the other way round: why do thematic analysis (which is harder) when causal mapping can identify not only some of the important themes but also tell you how they influence one another?
• 1. ... what you really want to do in the end, especially if you are doing evaluation, is find out what causes what in the eyes of your stakeholders. Identifying static themes can be interesting but often it's the causal information which helps you answer your main research and evaluation questions. Causal mapping is often a great way to cut to the chase.
• 1. Thematic analysis is more of an art than a science. "What are the main themes here" is a very open-ended question which can (and should) be interpreted in different ways by different analysts ("positionality"). Whereas people (and GPT-4) tend understand the instruction "identify each and every section of text which says that one thing causally influences another" quickly and easily, and they tend to agree on how to apply the rule.
• 1. The way we do causal mapping means identifying each and every causal connection. There is less room for someone's opinion in selecting what themes are most salient. Surprisingly, we can get good results without even a codebook of suggested themes aka causal factors, let alone bothering to train the AI or give it examples.
• 1. This means that the steps from your initial research idea all the way up to (but not including) your final analyses can be quite easily automated in a transparent way. You can train an army of analysts to the coding for you manually, or you can press the AI button, or a combination of both, and either way you will get pretty similar results. There isn't so much room for the opinion of your analysts, whether human or robot, at any point in the pipeline.
• Of course causal mapping is not free of bias (or positionality) due to human analysts' or AI-analysts' "world-views". It just leaves less room for those biases than more general thematic coding.
• And of course, there are times when general thematic analysis or some other kind of QDA is really what you need. We're just saying that causal mapping might fit your need more often than you think.